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Goal 


■ Review pros/cons of various playtest 
methodologies 

■ Discuss which data is best derived from which 
methodology 

■ Focus more research on user research 
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Overview 


■ Valve's (external) playtest philosophy 

■ Traditional playtest methodologies 
n Qualitative 

■ Technical playtest methodologies 
a Measured 
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Overview 


• Traditional Playtest Methodologies 

Direct Observation 
Verbal Reports 
Q&As 

• Technical Playtest Methodologies 

Stat Collection/Data Analysis 
Design Experiments 
Surveys 

Physiological Measurements 
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Valve's Game Design Process 


Goal is a fun game -> 

Game designs are hypotheses^ 
Playtests are experiments 
Evaluate designs off playtest results-> 
Repeat 
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Playtesting Goal 


■ Fun 

■ 

■ 

■ 


Ancillary Benefits 


■ Idea generation 

■ Identify problem areas 

■ Solve design arguments 

■ Aid other production aspects 
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Valve's Philosophy 


■ We want to make informed decisions 
a Get data early, get data often 

a Iterate constantly 

■ We don't know what's best (players do) 

■ Create a feedback loop between design and 
playtest 
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Content Creation 
+ 

Game Design 




Valve's Philosophy 


■ Playtesting continues after we ship 
a Gameplay stats 

a Forum responses 
a Fan feedback 

■ Always gathering data for the future 
a Patches/updates 

a Upcoming games 
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Traditional Methods 


■ Direct Observation 

■ Verbal Reports 

■ Q&As 
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Direct Observation 
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Direct Observation 


■ "Typical" playtest 

a Watch people play the game 
a Observe their gameplay/behavior 
a Simulate at-home experience 

■ Have a design goal 
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Direct Observation 


+ Get a feel for player interaction with game 

+ Importance of what people do—not what they 
say 

Presence of observers can bias results 
Salient event can slant interpretation 
Behavior requires interpretation 
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Verbal Reports 
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Verbal Reports 


■ Think-aloud protocol: 

a People describe their actions as they play 
a Unprompted and uncorrected 

■ In conjunction with direct observation 
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Verbal Reports 


+ Enables realtime glimpse into player 
thoughts, feelings, and motivations 

+ Bring up unnoticed details 
+ Effective for 'why' questions 

Interfere with gameplay/create an artificial 
experience/distracting 

Inaccurate and biased 
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Q&A 


■ Structured (usually) querying of playtesters 

■ Validate playtest goals 

■ Source of supplemental information 
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Q&A 


+ Answer specific design questions 
+ Determine specific player intent 

Group biases (anchoring, social pressure, 
saliency, etc.) 

People don't know why they do what they do 
Potential for biased questions 
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Our Q&A Procedure 


■ Survey 

■ Individual Q&A 

■ Group Q&A 

■ Be cautious 
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Benefits of Traditional 
Methods 

+ Nothing beats direct gameplay observation 

+ Determine major gameplay, navigation, and 
content issues 

+ Get an idea of player thoughts/mental models 
+ Get feedback on design choices 
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Issues with Traditional 
Methods 

Artificial gameplay sessions 

Many potential biases 

Distorted data (interpreted behavior) 

Lack of empiricism 

Missing elements of objectivity 

Sometimes difficult to establish emotions, 
baselines, and independence 
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Technical Approaches 


■ Stat collection/analysis 

■ Design experiments 

■ Surveys 

■ Physiological measurements 
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Stat Collection/Analysis 


■ Record of gameplay behaviors 

Deaths, level times, friendly fire,... 

■ Objective measurements 

■ Aggregate perspective 

■ Quantify behavior 

■ Opportunity for analyses 
a T-tests 

n Regressions 
□ 
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Stat Collection/Analysis 


+ Objective notions of player behavior 

+ See global trends 

+ Readily enables comparisons, baseline 
establishment, and metric creation 

+ Track changes overtime 
Averages hide extreme examples 
Miss nuance (lacking context) 

Requires rigor 

Can see 'illusory' patterns 
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Design Experiments 


■ Hypothesis testing 

a Compare two or more conditions 
a Collect data 
a Verify hypothesis 

■ Predict player behavior 

a Define set of variables 

a Investigate resulting relationships 
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jEflW! jOBTRtSS 2 a 

THE SCOUT UPDATE 


the results ARE IK 
the updates OVJ . 

NOW IT’S TIME TO... 


PLAY BALL! 


COMMUNITY VOTED UNLOCK ARLES ORDER 



1. THE FORCE-A-NATURE 

(REQUIRES 10 ACHIEVEMENTS TO UNLOCK) 


17.219 VOTES (42.53%) 


2. THE SANDMAN 

(REQUIRES IS ACHIEVEMENTS TO UNLOCK) 


13.906 VOTES (34.10%) 


3. RONK ENERGY DRINK 

(REQUIRES 30 AC1IIEVEMENTS TO UNLOCK) 


9.463 VOTES (23.37%) 
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Defeat logo, Counter-Strike, the Counter-Strike logo, 
lemaiks and/or registered trademarks of Valve Corporation. 
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Design Experiments 


+ Enables more informed decision-making 
+ Objective answer 
+ Saves time in the long run 
Costs time (in the short run) and money 
Right questions aren't always clear 
Proper experimental design is a process 
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Surveys 


■ Set of standardized questions 

■ Forced choice responses 

■ Quantify feedback/opinions 

■ Player categorization 
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How challenging were the following enemies (1 = very easy; 7 = very hard)? 


Boomer: 

Common Infected: 
Hunter: 

Smoker: 

Tank: 

Witch: 


1 2 
1 2 
1 2 
1 2 
1 2 
1 2 


3 4 
3 4 
3 4 
3 4 
3 4 
3 4 


5 6 
5 6 
5 6 
5 6 
5 6 
5 6 


7 

7 

7 

7 

7 

7 


Please rank order your preference for the following weapons from 
1 (most liked) to 12 (least liked) 


Assault Rifle 
Auto Shotgun 
Dual Pistols 
Gas Can 
Hunting Rifle 
Molotov Cocktail 
Mounted Turret 
Pipe Bomb 
Pistol 

Propane Tank 
Pump Shotgun 
SMG 



Simulated Questions 



Surveys 


+ Get less biased responses 
+ Validate responses (repetitive questions) 

+ Forced choice helpful for revealing preference 
+ Ratings enable time-based comparisons 
Eliminate nuance 

Difficulty in converting ratings to meaningful 
decisions 

Limited solution space 
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Physiological Measurements 


■ Measurements of biological response 

■ Create proxies of player state 

■ Involuntary 

■ Objective—can't be faked 

■ Quantify emotion 


IVA LV E I 



Valence and Arousal 


■ Valence = positive or negative emotion 

■ Arousal = magnitude of emotion 


IVA LV E I 






Heartrate 


■ Beat to beat interval 

■ Measure baseline rate and changes 

■ Most basic measure of arousal 

■ Fourier transforms to distinguish emotion 



http ://en.wikipedia.org/wiki/File:EKG2.png 
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Heartrate 


+ Simple to collect 
+ Accurate correlate of arousal 
+ Good metric for comparison 
Intrusive 

(Sometimes) delayed response to stimuli 
Variable 
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Skin Conductance Level 


■ Electrical resistance of the skin 
a Correlate with arousal 
a Maybe other emotions as well 


■ Can look for spikes (both responsive and 
anticipatory) 










Skin Conductance Level 


+ Excellent correlate with arousal 
+ Good metric for comparison 
+ Adept at detecting transient responses 
Intrusive 

Susceptible to other factors 
Direct I/O relationship doesn't exist 
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Eyetracking 


■ Camera focused on the eyes 

■ Determine where the eyes are looking 

■ Real-time insight into playerthought 
processes 

■ Blink rate/pupil dilation 
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Eyetracking 


+ Effective metric of player attention/gaze 
+ Excellent tool for interface design 

+ Provides understanding of scene 
interpretation 

Expensive 

Can be intrusive 

Time consuming 

Can lead to costly over-analysis 
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Face Recording 


■ Observation of facial expression 

■ Determination of player emotion 

■ Tied into gameplay 



T 
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Face Recording 


+ Provides emotional context 
+ Excellent metric of player emotion 
Intrusive 

Requires experienced coders 
Not always reliable 
Biased reactions 
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EEG 


■ Measurement of electrical potentials in the 
brain 

■ Various frequencies are correlated with 
emotional state 

n Alpha (relaxation) 
n Beta (thinking, engagement) 
n Delta (fatigue) 






EEG 

+ Good at measuring arousal, engagement, etc. 

+ Potential for fairly sophisticated 
determinations down the road 

Expensive 
Very intrusive 
Noisy 

Hard to control/validate 
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■ Sensors placed at varying points on the face 

■ Measurement of facial muscle 
contraction/relaxation 

■ Determinant of emotion based on 'action 
units' 



EMG 


+ Most accurate measure of emotion 
+ Real-time determination 
Expensive 
Very intrusive 


IVA LV E I 



Other Techniques 


■ Body temperature 

■ Gesture recognition 

■ Muscle tension 



Physiological Measurements 


+ More objective measurements of player state 
+ Quantifiable emotional response 
+ Analysis/comparison metrics 
Expensive 
Intrusive 

Artificial experience 
Requires experimental control 
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Benefits of Technical 
Approaches 

+ Application of empirical data to game design 
+ Objective (for the most part) 

+ Quantify behavior 

+ Enable testable hypotheses about player 
emotional state 
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Issues with Technical 
Approaches 


Expensive 
Resource intensive 
Impractical 
Lacking nuance 
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Summary 


■ Do your QA early 

■ Understand pros/cons of existing methods 

■ Correctly frame design questions 

■ Be aware of emerging technologies 
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